Skip to main content

View Post [edit]

Poster: got_1234 Date: Feb 21, 2017 1:35pm
Forum: faqs Subject: block archive bots

I which to block bots an remove my website

Se I send an email asking removal.

But for blocking i add some line on robots.txt and then tried to "Save Page Now"
but your bots still reaching
User-agent: ia_archiver
Disallow: /
User-agent: archive.org_bot
Disallow: /
User-agent: ia_archiver-web.archive.org
Disallow: /
User-agent: Alexa
User-agent: Archive.org
Disallow: /

Reply [edit]

Poster: got_1234 Date: Feb 21, 2017 1:41pm
Forum: faqs Subject: Re: block archive bots

And when i check log

wwwb-app11.us.archive.org - - [21/Feb/2017:11:11:11 +0100] "GET /a.html HTTP/1.1" 200 116 "" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0"


without UA